CHAPTER 6. Projection Methods. Let A R n n. Solve Ax = f. Find an approximate solution ˆx K such that r = f Aˆx L.

Transcription

1 Projection Methods CHAPTER 6 Let A R n n. Solve Ax = f. Find an approximate solution ˆx K such that r = f Aˆx L. V (n m) = [v, v 2,..., v m ] basis of K W (n m) = [w, w 2,..., w m ] basis of L Let x 0 be an initial iterate ˆx = x 0 + V y ˆr = f Aˆx = f A(x 0 + V y) = r 0 AV y. From W Tˆr = 0, or W T (r 0 AV y) = 0, we have the system (W T AV )y = W T r 0 which we need to solve for y. Thus, Scheme: Do until convergence select a pair of subspaces K and L choose bases V and W r = f Ax y = (W T AV ) W T r x := x + V y EndDo ˆx = x 0 + V (W T AV ) W T r 0. Note that the approximate solution is defined only when (W T AV ) is nonsingular, which is not guaranteed for any K and L even if A is nonsingular.

2 Example: Let A = 0 I m I m I m. If V = W = (e, e 2,..., e m ) = I m, then 0 W T AV = (I m, 0) 0 I m I m I m = (0, I m ) I m = 0. 0 I m 0 Theorem: Let A, K, and L satisfy either one of the following two conditions: (a) A is s.p.d. and L = K. (b) A is nonsingular and L = AK. Then B = W T AV is nonsingular for any bases V and W of K and L. Proof: (a) A := s.p.d. and L K W = V G nonsingular B = W T AV = G T V T AV }{{} s.p.d. nonsingular B is nonsingular (b) L = AK W = AV G B = W T AV = G T V T (A T A)V }{{} s.p.d. B is nonsingular 2

3 Two Optimality Results () A is s.p.d. ; W = V G where where x is the exact solution of Ax = f. Proof: E(x) = (x T x T )(Ax Ax) = (x T x T )(f Ax). E(ˆx) = min ˆx x 0 +K E(x) E 2 (x) = (x x) T A(x x) = x x 2 A Thus, for ˆx to be a minimizer of E(x) it is necessary and sufficient that V T ˆr = 0. Let d 0 = x x 0, and ˆd = x ˆx, where ˆx = x 0 + x. Thus, and i.e., (d 0 x, v) A = 0 for all v K. Aˆd = Ax Aˆx = A(d 0 x) = f Aˆx = ˆr, V T ˆr = V T Aˆd = V T A(d 0 x) = 0, d 0 dˆ. x A-orthogonal projection Note: ˆx = x 0 + x ˆd = x ˆx Aˆd = A(d 0 x) = ˆr 3

4 E(ˆx) = ˆd T Aˆd ; E(x 0 ) = d T 0 Ad 0 ˆd 2 A = ˆd T Aˆd = (d 0 x) T A(d 0 x) = d 0 2 A + xt A x 2 x T Ad 0 = d 0 2 A xt A x + 2[ x T A x x T Ad 0 ] = d 0 2 A xt A x + 2 x T A( x d 0 ) }{{} } ˆr {{ } zero (2) A is nonsingular ; L = AK, i.e., W = AV G ; G := nonsingular where ν(x) = f Ax 2. Proof: ν(ˆx) = min ν(x) x x 0 +K r 0 rˆ residual projection methods. A x AK V T A T ˆr = 0 ; (ˆr L) ˆr = f A(x 0 + x) = r 0 A x = (I P AK )r 0 ; ˆr 2 r 0 2. One-Dimensional Projection Methods K = span{v} L = span{w} 4

5 Scheme: Do until convergence r = f Ax α = w T r/(w T Av) x := x + αv EndDo (a) Steepest descent algorithm A := s.p.d. v = w = r Do until convergence r = f Ax α = r 2 2 /(rt Ar) x := x + αr EndDo Each step in the above algorithm minimizes, Scheme: r = f Ax p = Ar Do until convergence α = r T r/p T r x := x + αr r := r αp p = Ar EndDo φ(x) = x x 2 A = (x x) T A(x x). Theorem: For the steepest descent scheme ( ) λmax λ min d k+ A λ max + λ min 5 d k A

6 where, d k = x x k, and λ min λ(a) λ max. Proof: d k+ 2 A = dt k+ Ad k+. But, d k+ = x x k+ = x (x k + α k r k ) i.e., d k+ = d k α k r k. Also, Ad k = A(x x k ) = f Ax k = r k, and d k = A r k. Consequently, Thus, d k+ 2 A = dt k+ A(d k α k r k ) = d T k+ r k α k r T k+ r k. }{{} zero But, Hence, or Note: d k+ 2 A = (d k α k r k ) T r k = d T k r k α k r T k r k = r T k A r k α k r T k r k. α k = (r T k r k)/(r T k Ar k). d k+ 2 A = rt k A r k (rt k r k) 2 [(r T k Ar k) ] = (r T r k 4 k A 2 r k ) (r T k A r k )(r T k Ar, k) d k+ 2 A = d k 2 A [ r k 4 2 (r T k A r k )(r T k Ar k) r T k A r k = r T k A AA r k = d T k Ad k. Now we resort to the Kantorovich inequality which states that for any x 0, (x T x) 2 (x T Ax)(x T A x) 4λ maxλ min (λ max + λ min ) 2. 6 ].

7 As a result, [ d k+ 2 A d k 2 A 4λ ] maxλ min (λ max + λ min ) 2 [ ] = d k 2 (λmax λ min ) 2 A. (λ max + λ min ) 2 Q.E.D. Kantorovich Inequality Let A be s.p.d., then (r T r) 2 (r T Ar)(r T A r) 4λ λ n (λ + λ n ) 2 where λ(a): λ λ 2 λ n. λ 0 Proof: Let Q T AQ = Λ =..., then 0 λ n σ(r) = = (r T r) 2 (r T Ar)(r T A r) = (r T QQ T r) 2 (r T Q Q T r)(r T Q Q T r) (s T s) 2 (s T s) 2 (s T s) ; s = QT r i.e., where, σ(r) = α(r) = 0 i= = α(r) β(r) 0 n λ i s 2 i n 2 s 2 C i A i= 0 CB A@ n i= ; θ n i = λ i θ i i= λ i s 2 i n i= s 2 C i A s 2 i 7

8 -- λ β* α λ n λ λ λ α = λ λ n β* λ λ λ n λ Therefore, β(r) = β = λ + = σ(r) = α(r) β(r) n i= λ i θ i ; θ i [ ] ( λ λ λ λ n λ n λ λ λ n (λ + λ n λ). min λ n λ λ ) (/λ) [(λ + λ n λ)/λ λ n ] = min λ d dλ [λ(λ + λ n ) λ 2 ] = (λ + λ n ) 2λ = 0 λ = 2 (λ + λ n ); d 2 ( ) = 2 < 0 dλ2 λ λ n σ(r) (λ 4 + λ n ) = 4λ λ 2 2 (λ + λ n ). 2 8 λ λ n [λ(λ + λ n ) λ 2 ]

9 Note: Q.E.D. [ d k+ 2 A λmax λ min λ max + λ min ( κ2 (A) = κ 2 (A) + where κ 2 (A) = λ max (A)/λ min (A). (Here A is s.p.d.) Minimal Residual Iteration ] 2 d k 2 A ) 2 d k 2 A A one-dimensional projection process for A nonsymmetric but with a s.p.d. symmetric part, i.e., 2 (A + AT ) is s.p.d. K = span{v} v = r L = span{w} w = Ar Scheme: Until convergence, Do: r = f Ax α = (r T Ar)/(r T A T Ar) x := x + αr Each step minimizes f Ax 2 2 in the relaxation direction r. Theorem: Let A s = 2 (A + AT ) be s.p.d., and let σ = A 2 = ρ /2 (A T A) µ = λ min (A s ), then ] r k+ 2 2 [ µ2 σ 2 r k 2 2 and the minimal residual iteration converges for any initial iterate x 0. 9

10 Proof: r k+ 2 2 = f A(x k + α k r k ) 2 2 = r k α k Ar k 2 2 = (r T k+ r k) α k (r T k+ Ar k) 0 r k+ L = r T k r k α k r T k Ar k Consider the eigenvalue problem = r k 2 2 (rt k Ar k) 2 (r T k AT Ar k ) = r k 2 2 [ (rt k Ar k) 2 r k 4 2 r k 2 2 Ar k 2 2 ]. where A ss = (A A T )/2, then Au = λu ; λ = ut Au u T u = ut (A s + A ss )u, u T u r k+ 2 2 r k 2 2 [ ] u σ 2 λ 2 T 2 A s u = λ 2 u T u min(a s ). [ λ 2 min (A s) r ] k 2 2. But Ar k 2 2 Ar k 2 2 A 2 2 r k 2 2 r k 2 2 = Ar k 2 2 A 2 2 σ. 2 0

11 Thus, Note that σ 2 > µ 2. [ r k+ 2 2 r k 2 2 λ2 min (A ] s) ) σ 2 = ( µ2 r k 2 2. σ 2 Q.E.D. Observation: From but r k+ 2 2 = r k 2 2 (rt k Ar k) 2 (r T k AT Ar k ) [ = r k 2 2 (rt k Ar k) (r T k AT Ar k ) rt k Ar ] k r T k r k r T k cos γ k = (Ar k) Ar k 2 r k 2 acute angle bet. r k & Ar k r k+ 2 2 = r k 2 2 [ cos2 γ k ] = r k 2 2 sin2 γ k. MR-iteration: r = f Ax p = Ar Do until convergence α = (r T p)/(p T p) x := x + αr r := r αp p = Ar EndDo Note: r k+ = r k α k p k.

12 Residual Norm Steepest Descent A := nonsingular (A + A T ) need not be s.p.d. K = span{v} v = A T r L = span{w} w = Ar Scheme: matrics 2 r = f Ax ; v = A T r α = (v T v)/(v T A T Av) = v 2 2 / Av x := x + αv Can we reduce the number of matrix-vector products? Hint: r new = f A(x + αv) = r αav. Modified scheme: r = f Ax. Do until convergence v := A T r z := Av α := v 2 2 / z 2 2 x := x + αv r := r αz EndDo 2